Extending Huffman Coding for Multilingual Text Compression

Traditional text compression algorithms such as Huffman and LZ variants are usually based on 8-bit characters sampling. However, under the unicode representation for multilingual information, the character set of each language such as Chinese and Japanese is consisted of a very number of distinct characters and thus 16-bit or 32-bit character sampling is needed. Consequently, when text compression algorithms based on 8-bit character sampling’is applied to documents using 16-bit or 32 bit character sampling, very poor data compression ratio (average about 1.5) is obtained. In this paper, we propose two new algorithms that are based on the 16-bit or 32-bit sampling character set and on the unique features of the languages with large number of distinct characters to improve data compression ratios for multilingual text documents significantly. We choose Chinese language using 16 bit character sampling (such as Big-5 or GB code) as the representative language in our study. The first approach, called the Static Chinese Huffman Coding (Huffs &, is to introduce the concept of a single Chinese character in the Huffman tree. Experimental results on our PH corpus showed that the improvement in compression ratio obtained by ff@-s chi ranges from 20% to 29%. The second approach, called the Dictionary-Based Chinese Huffman Coding